Make vote timestamps optional (ref #5016) #5122

Nutomic · 2024-10-17T09:18:37Z

Turns out timestamps and comment_like.post_id arent used at all in the backend. So this PR adds a setting database.store_vote_timestamps: bool to avoid storing them (default false). Also comment_like.post_id is removed. If timestamps are enabled, they are now returned by the api in like list endpoints. With the setting disabled, storage usage changes like this:

Previous size of post_like column: 4 + 4 + 2 + 8 = 18 bytes
Previous size of comment_like: 4 + 4 + 2 + 8 + 4 = 22 bytes

After this change both are 10 bytes so the table size should also be reduzed by half roughly, so in case of lemmy.ml from 15 gb to ~7.5 gb. With 40gb total db size this is a reduction by 19%.

Nothing4You · 2024-10-17T09:30:53Z

If this isn't used, how does Lemmy deal with receiving vote activities out of order?
For example, someone upvotes a post, then downvotes a post, but due to parallel sending and a random connection failure the downvote gets sent first, then the upvote gets sent. The result should still show the user having downvoted the post on remote instances.

I mentioned this in the other issue already, would it save a meaningful amount of space if the column was nullable and a scheduled task sets e.g. all votes older than a week to published at NULL?

Nutomic · 2024-10-17T09:38:38Z

The timestamp for outgoing activities is generated in a different place, and doesnt use these columns at all.

Nothing4You · 2024-10-17T09:51:30Z

I'm talking about the receiving instance.
For example, lemmy.ml sends these activities out of order to lemm.ee, how would lemm.ee know to drop the older but later received vote?
Given that there could be some safeguards within Lemmy on the sending side, the same issue would exist with other apub software that has asynchronous parallel sending of activities.

Nutomic · 2024-10-17T09:58:53Z

At the moment there is no logic on the receiving side to reorder incoming activities. So we trust that sending happens in the correct order (which is true for Lemmy).

I attempted to implement this in #4559 but it was too complicated.

Pdzly · 2024-10-17T09:58:55Z

Its used in vote abuse tracking

phiresky · 2024-10-17T11:23:51Z

just to mention, afaik deleting a column doesn't actually free any space for existing databases, and the methods to do so require full exclusive lock (vacuum full) https://dba.stackexchange.com/questions/117510/reclaim-disk-space-from-dropped-column-without-downtime . doesn't mean this is a bad idea though and I'm not sure about new rows

Nutomic · 2024-10-17T12:11:34Z

We can mention in the release notes about running vacuum to reclaim disk space. Or how about changing the columns to nullable and setting all values to null before dropping? That should theoretically reclaim most of the space.

Edit: Added this now.

Illecors · 2024-10-17T12:24:50Z

Space is reclaimed by postgres, but without a full vacuum the OS will not be notified about it. In general - that's not an issue as postgres will reuse that space if and when necessary for other things.

asimons04 · 2024-10-17T12:28:55Z

Its used in vote abuse tracking

Can concur. There was a vote manipulation campaign recently, and having the timestamps available for analysis proved very helpful in identifying participating accounts.

I was actually hoping for the opposite of this PR: the vote timestamps being added to the API response so I didn't have to always run analysis against the database.

Nutomic · 2024-10-17T13:57:22Z

Updated, see op for details.

dessalines · 2024-10-17T13:58:10Z

In that case then maybe lets just keep the timestamp columns and only remove the post_id one. We'll have to think of other ways to save space, like making these boolean instead of smallint (2B -> 1B). But that could be done later.

dessalines · 2024-10-17T14:01:36Z

Turns out timestamps and comment_like.post_id arent used at all in the backend. So this PR adds a setting database.store_vote_timestamps: bool

I'm not a big fan of that solution, because timestamps affect every table equally, and I think we should just accept that its a space hit we can't avoid. Could get increasingly complicated if we start adding special rules for specific tables. There's gotta be other ways to deal with the space issue.

Nutomic · 2024-10-17T14:24:55Z

There is nothing complicated, it just checks the setting before writing a vote, and either sets it to the current time or to null. And the column isnt used anywhere in the backend so the actual value doesnt matter for us. Most instances wont enable the setting, and will benefit from 20% lower db size which is very significant.

dessalines · 2024-10-17T14:41:05Z

I won't be a blocker for adding it then, but I'd like @phiresky @dullbananas and @SleeplessOne1917 to chime in too.

dullbananas · 2024-10-17T16:32:05Z

Timestamp shouldn't be optional.

There's some possible ways to reduce the storage space of the timestamp, all of which will be more effective if it's replaced with the duration between the post or comment timestamp and the vote:

Split into two 32 bit integers, and make one of them optional
Use an extension that provides column compression (not timescaledb because that's pooprietary)
Compression in the file system

Nutomic · 2024-10-18T08:06:52Z

Whats wrong with an optional timestamp if its not required anywhere in the code? We can rename it to published_opt to make it completely clear that its not a normal timestamp. Its really a minor problem considering that it reduces db size by 1/5th.

Using some kind of manual compression would be too complicated, and would not allow sql queries to find abusive votes. From what I can find postgres doesnt have any compression which would help in this case.

Nothing4You · 2024-10-19T01:47:01Z

There is actually a place where vote timestamps are used in Lemmy: community and site aggregates for active user counting

lemmy/migrations/2023-12-06-180359_edit_active_users/up.sql

Lines 35 to 55 in 859dfb3

    
           SELECT 
        
               pl.person_id, 
        
               p.community_id 
        
           FROM 
        
               post_like pl 
        
               INNER JOIN post p ON pl.post_id = p.id 
        
               INNER JOIN person pe ON pl.person_id = pe.id 
        
           WHERE 
        
               pl.published > ('now'::timestamp - i::interval) 
        
               AND pe.bot_account = FALSE 
        
           UNION 
        
           SELECT 
        
               cl.person_id, 
        
               p.community_id 
        
           FROM 
        
               comment_like cl 
        
               INNER JOIN post p ON cl.post_id = p.id 
        
               INNER JOIN person pe ON cl.person_id = pe.id 
        
           WHERE 
        
               cl.published > ('now'::timestamp - i::interval) 
        
               AND pe.bot_account = FALSE) a

lemmy/migrations/2023-12-06-180359_edit_active_users/up.sql

Lines 92 to 110 in 859dfb3

    
           SELECT 
        
               pl.person_id 
        
           FROM 
        
               post_like pl 
        
               INNER JOIN person pe ON pl.person_id = pe.id 
        
           WHERE 
        
               pl.published > ('now'::timestamp - i::interval) 
        
               AND pe.local = TRUE 
        
               AND pe.bot_account = FALSE 
        
           UNION 
        
           SELECT 
        
               cl.person_id 
        
           FROM 
        
               comment_like cl 
        
               INNER JOIN person pe ON cl.person_id = pe.id 
        
           WHERE 
        
               cl.published > ('now'::timestamp - i::interval) 
        
               AND pe.local = TRUE 
        
               AND pe.bot_account = FALSE) a;

Nutomic · 2024-10-21T12:37:17Z

@Nothing4You Youre right, then this is not really an option. Unless we only delete timestamps for votes which are older than six months, but thats probably not worth the effort.

dessalines · 2024-10-22T20:16:48Z

Can you re-open this one and just do the post_id column removal? The other PR got closed, but its migration was also incorrect.

I can do this if you like.

Nutomic · 2024-10-23T07:58:06Z

#5134

…5134)

* Add option to search exclusively by post title (#5015) * Add option to search exclusively by post title * Address format issues * Remove duplicated 'removed' filter * Replace url_search with search_term * Build generic PostQuery before search match * Create default queries. Move title_only to Search struct. Rename Url to PostURL * Revert PostUrl to Url * Upgrading webmention to 0.6.0, removes native-tls (#4976) * Removing embedded pict-rs. (#5023) Some reasons for removing this: - Even as an optional dependency, it locks us to many specific versions of rust deps. - Pict-rs is a large app that can and should be run in on its own. - Violates the philosophy of separation of concerns. * Adding clearurls crate to clean tracking params from links and markdown. (#5018) * Adding clearurls crate to clean tracking params from links and markdown. - Thanks to @jenrdikw for creating this - Fixes #4905 * Upgrading to new version of clearurls * Fix clippy * Remove enable nsfw (#5017) * Remove `local_site.enable_nsfw` in favor of `site.content_warning` (fixes #4627) * cleanup usage of SiteView::read_local * test * uppercase * SSO Support (#4881) * Added OAUTH2 OIDC support * Fixes and improvements based on review feedback * use derive_new::new instead of TypedBuilder * merge migrations into a single file * fixes based on review feedback * remove unnecessary hostname_ui config * improvement based on review feedback * improvements based on review feedback * delete user oauth accounts at account deletion * fixes and improvements based on review feedback * removed auto_approve_application * support registration application with sso * improvements based on review feedback * making the TokenResponse an internal struct as it should be * remove duplicate struct * prevent oauth linking to unverified accounts * switched to manually entered username and removed the oauth name claim * fix cargo fmt * fix compile error * improvements based on review feedback * fixes and improvements based on review feedback --------- Co-authored-by: privacyguard <[email protected]> * Adding ability to restore content on user unban. (#4845) * Adding ability to restore content on user unban. - Fixes #4721 * Fixing api tests. * Fix package.json * Fixing lemmy-js-client dep. * Adding API test for restoring content. * Adding a default_comment_sort_type column for local_site and local_user. (#4469) * Adding a default_comment_sort_type column for local_site and local_user. - Renamed SortType to PostSortType in the DB and code. - Renamed references to default_sort_type to default_post_sort_type. - Fixes #4128 * Renaming migration to current date. * Simplifying PostSortType. * Simplify tests using default (#5026) * Feature/custom emoji and tagline views (#4580) * Add custom_emoji list route * Add tagline list route * Apply linting * Remove unecessary TaglineView * Add category filter for custom emoji * Add create tagline endpoint * Add update tagline endpoint * Add delete tagline endpoint * Format through lint.sh * Remove custom_emojis and taglines from site resource * Get random tagline on site requets * Impl Crud for Tagline Remove superfluous properties * Move tagline endpoints under /admin * Impl Crud for CustomEmoji * Remove delete from tagline and custom emoji impls * Check markdown for tagline * Validate markdown on tagline * Make content fields non optional Add error types for tagline validation * Use process_markdown instead of process_markdown_opt * Consolidate Tagline error types * Remove unecessary clone * Updat misleading comments * Remove local_site_id from tagline and custom_emoji * Update TaglineInserForm and TaglineUpdateForm * Add ignore_page_limits for custom emojis EmojiPicker needs to be able to retrieve all emojis in 1 call * Update custom_emoji_view Only keep get_all als helper function calling list with paging ignored Only order on category when filtering on category * Removing pointless get_all fn. * remove tagline length checks * make fields of TaglineInsertForm and TaglineUpdateForm mandatory * move emoji order statement * add comment for GetSiteResponse.tagline --------- Co-authored-by: Freakazoid182 <> Co-authored-by: SleeplessOne1917 <[email protected]> Co-authored-by: Dessalines <[email protected]> Co-authored-by: Felix Ableitner <[email protected]> * Add category to RSS feeds (fixes #3446) (#5030) * Unittest for Search by title only (#5033) * added test for search by title only * formatted rust files * Upgrading to rust 1.81 (#5032) * Remove TypedBuilder in favor of derive_new (fixes #4863) (#5020) * Remove TypedBuilder in favor of derive_new (fixes #4863) * fix * fix * Ignore zero values when setting rate limits (fixes #4280) (#5029) * Ignore zero values when setting rate limits (fixes #4280) Havent bothered to add an error message for such an uncommon case. * fmt * reorder, add test * Always save remote image data (#4875) * Always save remote image data * cleanup --------- Co-authored-by: Felix Ableitner <[email protected]> Co-authored-by: Dessalines <[email protected]> * Get rid of a lot of pointless mut form initializations. (#5037) * Get rid of a lot of pointless mut form initializations. - Fixes #5036 * Fix clippy. * Simplify handling of NotFound SQL errors (fixes #4633) (#5031) * Simplify handling of NotFound SQL errors (fixes #4633) * fmt * wip * compiling * clippy * api tests * fix * Adding saved_only, liked_only, and disliked_only filters to search. (#5034) * Adding saved_only, liked_only, and disliked_only filters to search. - Fixes #4547 * Removing duplicate Url return type for search (was actually post). - This now works like the post_title_only filter. * Address PR comments. * Add saved_only post_view test. * Removing a few more Result<bool> . (#4977) * Removing a few more Result<bool> . * Running taplo fmt. * Running fmt. * Adding email taken test. * Fixing tests. * Adding back in missing admin check. * Rename check_has_local_followers function. * Conditionally hide comments on nsfw posts (fixes #4237) (#5028) * Conditionally hide comments on nsfw posts (fixes #4237) * fix test * Post scheduling (fixes #234) (#5025) * Post scheduling (fixes #234) * clippy * replace map_err with inspect_err * ignore unpublished posts in read queries * add api test * fmt * add some checks * address some review comments * allow updating schedule time * rewrite scheduled task * fmt * machete * compare date in sql, more filters * check for community ban in sql * remove api test (scheduled task only runs every 10 mins) * remove mut * add index * remove Post::read impl * fmt * fix * correctly handle changes to schedule time * normal users can only schedule up to 10 posts * Remove redundant local_user.auto_expand setting. (#5041) - Fixes #4643 Co-authored-by: SleeplessOne1917 <[email protected]> * Add ability to search for Community by its description (or title only). (#5044) - This changes the post_title_only for Search to title_only, since its also used in the community query now. - Fixes #4785 * Cleanup remaining use of Result<bool, Error> (fixes #4862) (#5047) * Replace clippy allow annotation with expect (fixes #5012) (#5048) * Add skip_serialize_none to OAuth structs with option fields (#5046) * Add skip_serialize_none to OAuth structs with option fields * PR feedback * Remove serde and ts export from SSO db-only structs * Apply scheduled post limit to future posts instead of past posts, and verify this in test (#5054) * test scheduled_post_count * fix syntax error * fix formatting * fix argument order * fix user_scheduled_post_count function * Avoid breaking changes, keep response fields as deprecated (#5058) * Adding skip_serializing_none to another OAuth API request. (#5060) * Handle partial settings backup (fixes #4307) (#5063) * Handle partial settings backup (fixes #4307) * clippy * Avoid stack overflow when fetching nested comments, reduce max comment depth to 50 (#5009) * Avoid stack overflow when fetching deeply nested comments * add test case * reduce comment depth, add docs * decrease * reduce max comment depth to 50 * fmt * clippy * cleanup * Update Rust crate clap to v4.5.18 (#5066) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * Update Rust crate reqwest to v0.12.8 (#5068) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * Update Rust crate sitemap-rs to v0.2.2 (#5069) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * Update Rust crate tracing-actix-web to v0.7.13 (#5070) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * Changing renovate to run on the weekends. (#5067) - Lemmy builds take a long time, and the current schedule only runs for about 4 hours once a month, and renovate also rate-limits itself to ~ 2 per hour. This gives it a large enough window, without clogging up my runner machines. * Update Rust crate async-trait to v0.1.83 (#5065) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * Update dependency @types/jest to v29.5.13 (#5071) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * Attempt to fix test for fetching deeply nested comment (#5072) Co-authored-by: Dessalines <[email protected]> * Add modlog entries for bulk removals. (#5050) * Add modlog entries for bulk removals. - Added unit tests for removal / restore to api_common/utils. - Fixes #4699 * Address PR comments. * Combining remove and restore functions. * Trigger build. * Trigger build 2. * Changing allow to expect. * Adding local site settings to reject federated upvotes or downvotes. (#5038) * Adding local site settings to reject federated upvotes or downvotes. - Should help defend against downvote spamming instances. - Fixes #4086 * Adding new vote mode types. * Simpler activitypub vote check. * Adding undo vote for failed vote mode check. * Update crates/api_common/src/utils.rs --------- Co-authored-by: Nutomic <[email protected]> * Replace `clippy::unwrap_used` in tests (#5064) * Add LemmyResult to session_middleware tests * Add LemmyResult to inboxes tests * Add LemmyResult to slurs tests * Add LemmyResult to markdown tests * Add LemmyResult to rate_limiter tests * Add LemmyResult to error tests * Add LemmyResult to api_common utils tests * Add LemmyResult to request tests * Add LemmyResult to claims tests * Propagate registration_applications errors * Remove clippy::unwrap_used from community tests * Add LemmyResult to community_view tests * Add LemmyResult to db_schema post tests * Add LemmyResult to site_aggregates tests * Add LemmyResult to private_message tests * Add LemmyResult to activity tests * Add LemmyResult to federation_allowlist tests * Add LemmyResult to comment_aggregates tests * Add LemmyResult to post_report tests * Add LemmyResult to moderator tests * Add LemmyResult to community_aggregates tests * Add LemmyResult to person_aggregates tests * Add LemmyResult to language tests * Add LemmyResult to post_aggregates tests * Add LemmyResult to db_schema comment tests * Add LemmyResult to actor_language tests * Add LemmyResult to vote_view tests * Add LemmyResult to registration_application_view tests * Add LemmyResult to private_message_view tests * Add LemmyResult to private_message_report_view tests * Add LemmyResult to post_report_view tests * Add LemmyResult to comment_report_view tests * Add LemmyResult to sitemap tests * Replace .expect() with .unwrap() * Format code * Remove clippy::unwrap_used from activity tests * Add diesel result in db_schema tests * Format code * Map to_bytes() error to LemmyErrorType * Remove clippy::unwrap_used from error tests * Removing a few more unwraps, and cleaning up language code. * Replace map_err with unwrap_or_default * Replace ok_or with and_then --------- Co-authored-by: Dessalines <[email protected]> * Adding a get_random_community endpoint. (#5042) * Adding a get_random_community endpoint. - Fixes #4698 * Fixing issue from main. * Adding ListingType to the query. * More concise query filter. * Resolve links to remote posts into local URL (#5057) * move code to new file * rewrite markdown links (fixes #2987) * add missing file * add helper fn * also convert post.url * simplify search.rs * clippy * also rewrite user/community links in markdown * Call from apub handlers, cleanup * no network requests in test * clippy * fix tests * serial * test * no mut * add api test * fix api test * Update Rust crate clap to v4.5.19 (#5080) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * Update Rust crate html2text to v0.12.6 (#5081) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * Update Rust crate rustls to v0.23.14 (#5083) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * Update rust-futures monorepo to v0.3.31 (#5090) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * Update Rust crate actix-web-prom to 0.9.0 (#5091) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * Update Rust crate regex to v1.11.0 (#5093) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * Update Rust crate serde_with to v3.10.0 (#5094) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * Update Rust crate serde_with to v3.11.0 (#5095) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * Update dependency ts-jest to v29.2.5 (#5089) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * Update dependency typescript to v5.6.2 (#5098) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * Update dependency eslint to v9.12.0 (#5097) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * Update docker/dockerfile Docker tag to v1.10 (#5099) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * Update pnpm to v9.12.0 (#5100) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * Update Rust crate reqwest to v0.12.8 (#5082) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * Update dependency @types/node to v22.7.4 (#5096) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * Update tamasfe/taplo Docker tag to v0.9.3 (#5101) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * Update tmknom/prettier Docker tag to v3.2.5 (#5102) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * Add test case for community.hidden in post_view (ref #5074) (#5106) * Resize post thumbnails (fixes #4053) (#5107) * Resize post thumbnails (fixes #4053) * 256px * Update typescript-eslint monorepo to v8.8.1 (#5103) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * Update renovate config (#5109) * Update renovate config * ignroe rust updates, run on first day of month * fix * Add community alphabetic sorting (#5056) * Started * Finished? Need to write tests * Formatting * Formatting * Formatting * Write tests * Formatting * Formatting * Formatting * Unnecessary lifetime * Safety * Unwrap * Formatting * Formatting * Fix local_only test * Formatting * Name consistency * Adding lower to community name sort. --------- Co-authored-by: Dessalines <[email protected]> Co-authored-by: Dessalines <[email protected]> * Support markdown sub/superscript, use external crate for spoilers (#5135) * Use external crate for spoiler tags * Also add other plugins * fix test * Remove comment_like.post_id column which is unnecessary (ref #5122) (#5134) * Allow admins to resolve removed or deleted objects via API (#5061) * Allow admins to resolve removed or deleted objects via API * Removing pointless TestUser. --------- Co-authored-by: Dessalines <[email protected]> * Remove individual user/community inboxes (#5124) * Remove endpoints for individual community/user inboxes fixes #4147 fixes #3928 * Remove shared_inbox_url columns * fmt * Fixing errors. --------- Co-authored-by: Carlos Cabello <[email protected]> Co-authored-by: Nutomic <[email protected]> Co-authored-by: privacyguard <[email protected]> Co-authored-by: privacyguard <[email protected]> Co-authored-by: Freakazoid182 <[email protected]> Co-authored-by: SleeplessOne1917 <[email protected]> Co-authored-by: leoseg <[email protected]> Co-authored-by: Sander Saarend <[email protected]> Co-authored-by: Joseph Silva <[email protected]> Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> Co-authored-by: netbrum <[email protected]> Co-authored-by: Steven Vergenz <[email protected]> Co-authored-by: Richard Schwab <[email protected]>

…ct (#5024) * Add test case to ensure all errors are in use * Add test case to find errors that are only used for federation * Move federation errors into separate enum (fixes #4926) * clippy * clippy * remove serde tag * Reduce errors dess (#5138) * Add option to search exclusively by post title (#5015) * Add option to search exclusively by post title * Address format issues * Remove duplicated 'removed' filter * Replace url_search with search_term * Build generic PostQuery before search match * Create default queries. Move title_only to Search struct. Rename Url to PostURL * Revert PostUrl to Url * Upgrading webmention to 0.6.0, removes native-tls (#4976) * Removing embedded pict-rs. (#5023) Some reasons for removing this: - Even as an optional dependency, it locks us to many specific versions of rust deps. - Pict-rs is a large app that can and should be run in on its own. - Violates the philosophy of separation of concerns. * Adding clearurls crate to clean tracking params from links and markdown. (#5018) * Adding clearurls crate to clean tracking params from links and markdown. - Thanks to @jenrdikw for creating this - Fixes #4905 * Upgrading to new version of clearurls * Fix clippy * Remove enable nsfw (#5017) * Remove `local_site.enable_nsfw` in favor of `site.content_warning` (fixes #4627) * cleanup usage of SiteView::read_local * test * uppercase * SSO Support (#4881) * Added OAUTH2 OIDC support * Fixes and improvements based on review feedback * use derive_new::new instead of TypedBuilder * merge migrations into a single file * fixes based on review feedback * remove unnecessary hostname_ui config * improvement based on review feedback * improvements based on review feedback * delete user oauth accounts at account deletion * fixes and improvements based on review feedback * removed auto_approve_application * support registration application with sso * improvements based on review feedback * making the TokenResponse an internal struct as it should be * remove duplicate struct * prevent oauth linking to unverified accounts * switched to manually entered username and removed the oauth name claim * fix cargo fmt * fix compile error * improvements based on review feedback * fixes and improvements based on review feedback --------- Co-authored-by: privacyguard <[email protected]> * Adding ability to restore content on user unban. (#4845) * Adding ability to restore content on user unban. - Fixes #4721 * Fixing api tests. * Fix package.json * Fixing lemmy-js-client dep. * Adding API test for restoring content. * Adding a default_comment_sort_type column for local_site and local_user. (#4469) * Adding a default_comment_sort_type column for local_site and local_user. - Renamed SortType to PostSortType in the DB and code. - Renamed references to default_sort_type to default_post_sort_type. - Fixes #4128 * Renaming migration to current date. * Simplifying PostSortType. * Simplify tests using default (#5026) * Feature/custom emoji and tagline views (#4580) * Add custom_emoji list route * Add tagline list route * Apply linting * Remove unecessary TaglineView * Add category filter for custom emoji * Add create tagline endpoint * Add update tagline endpoint * Add delete tagline endpoint * Format through lint.sh * Remove custom_emojis and taglines from site resource * Get random tagline on site requets * Impl Crud for Tagline Remove superfluous properties * Move tagline endpoints under /admin * Impl Crud for CustomEmoji * Remove delete from tagline and custom emoji impls * Check markdown for tagline * Validate markdown on tagline * Make content fields non optional Add error types for tagline validation * Use process_markdown instead of process_markdown_opt * Consolidate Tagline error types * Remove unecessary clone * Updat misleading comments * Remove local_site_id from tagline and custom_emoji * Update TaglineInserForm and TaglineUpdateForm * Add ignore_page_limits for custom emojis EmojiPicker needs to be able to retrieve all emojis in 1 call * Update custom_emoji_view Only keep get_all als helper function calling list with paging ignored Only order on category when filtering on category * Removing pointless get_all fn. * remove tagline length checks * make fields of TaglineInsertForm and TaglineUpdateForm mandatory * move emoji order statement * add comment for GetSiteResponse.tagline --------- Co-authored-by: Freakazoid182 <> Co-authored-by: SleeplessOne1917 <[email protected]> Co-authored-by: Dessalines <[email protected]> Co-authored-by: Felix Ableitner <[email protected]> * Add category to RSS feeds (fixes #3446) (#5030) * Unittest for Search by title only (#5033) * added test for search by title only * formatted rust files * Upgrading to rust 1.81 (#5032) * Remove TypedBuilder in favor of derive_new (fixes #4863) (#5020) * Remove TypedBuilder in favor of derive_new (fixes #4863) * fix * fix * Ignore zero values when setting rate limits (fixes #4280) (#5029) * Ignore zero values when setting rate limits (fixes #4280) Havent bothered to add an error message for such an uncommon case. * fmt * reorder, add test * Always save remote image data (#4875) * Always save remote image data * cleanup --------- Co-authored-by: Felix Ableitner <[email protected]> Co-authored-by: Dessalines <[email protected]> * Get rid of a lot of pointless mut form initializations. (#5037) * Get rid of a lot of pointless mut form initializations. - Fixes #5036 * Fix clippy. * Simplify handling of NotFound SQL errors (fixes #4633) (#5031) * Simplify handling of NotFound SQL errors (fixes #4633) * fmt * wip * compiling * clippy * api tests * fix * Adding saved_only, liked_only, and disliked_only filters to search. (#5034) * Adding saved_only, liked_only, and disliked_only filters to search. - Fixes #4547 * Removing duplicate Url return type for search (was actually post). - This now works like the post_title_only filter. * Address PR comments. * Add saved_only post_view test. * Removing a few more Result<bool> . (#4977) * Removing a few more Result<bool> . * Running taplo fmt. * Running fmt. * Adding email taken test. * Fixing tests. * Adding back in missing admin check. * Rename check_has_local_followers function. * Conditionally hide comments on nsfw posts (fixes #4237) (#5028) * Conditionally hide comments on nsfw posts (fixes #4237) * fix test * Post scheduling (fixes #234) (#5025) * Post scheduling (fixes #234) * clippy * replace map_err with inspect_err * ignore unpublished posts in read queries * add api test * fmt * add some checks * address some review comments * allow updating schedule time * rewrite scheduled task * fmt * machete * compare date in sql, more filters * check for community ban in sql * remove api test (scheduled task only runs every 10 mins) * remove mut * add index * remove Post::read impl * fmt * fix * correctly handle changes to schedule time * normal users can only schedule up to 10 posts * Remove redundant local_user.auto_expand setting. (#5041) - Fixes #4643 Co-authored-by: SleeplessOne1917 <[email protected]> * Add ability to search for Community by its description (or title only). (#5044) - This changes the post_title_only for Search to title_only, since its also used in the community query now. - Fixes #4785 * Cleanup remaining use of Result<bool, Error> (fixes #4862) (#5047) * Replace clippy allow annotation with expect (fixes #5012) (#5048) * Add skip_serialize_none to OAuth structs with option fields (#5046) * Add skip_serialize_none to OAuth structs with option fields * PR feedback * Remove serde and ts export from SSO db-only structs * Apply scheduled post limit to future posts instead of past posts, and verify this in test (#5054) * test scheduled_post_count * fix syntax error * fix formatting * fix argument order * fix user_scheduled_post_count function * Avoid breaking changes, keep response fields as deprecated (#5058) * Adding skip_serializing_none to another OAuth API request. (#5060) * Handle partial settings backup (fixes #4307) (#5063) * Handle partial settings backup (fixes #4307) * clippy * Avoid stack overflow when fetching nested comments, reduce max comment depth to 50 (#5009) * Avoid stack overflow when fetching deeply nested comments * add test case * reduce comment depth, add docs * decrease * reduce max comment depth to 50 * fmt * clippy * cleanup * Update Rust crate clap to v4.5.18 (#5066) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * Update Rust crate reqwest to v0.12.8 (#5068) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * Update Rust crate sitemap-rs to v0.2.2 (#5069) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * Update Rust crate tracing-actix-web to v0.7.13 (#5070) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * Changing renovate to run on the weekends. (#5067) - Lemmy builds take a long time, and the current schedule only runs for about 4 hours once a month, and renovate also rate-limits itself to ~ 2 per hour. This gives it a large enough window, without clogging up my runner machines. * Update Rust crate async-trait to v0.1.83 (#5065) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * Update dependency @types/jest to v29.5.13 (#5071) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * Attempt to fix test for fetching deeply nested comment (#5072) Co-authored-by: Dessalines <[email protected]> * Add modlog entries for bulk removals. (#5050) * Add modlog entries for bulk removals. - Added unit tests for removal / restore to api_common/utils. - Fixes #4699 * Address PR comments. * Combining remove and restore functions. * Trigger build. * Trigger build 2. * Changing allow to expect. * Adding local site settings to reject federated upvotes or downvotes. (#5038) * Adding local site settings to reject federated upvotes or downvotes. - Should help defend against downvote spamming instances. - Fixes #4086 * Adding new vote mode types. * Simpler activitypub vote check. * Adding undo vote for failed vote mode check. * Update crates/api_common/src/utils.rs --------- Co-authored-by: Nutomic <[email protected]> * Replace `clippy::unwrap_used` in tests (#5064) * Add LemmyResult to session_middleware tests * Add LemmyResult to inboxes tests * Add LemmyResult to slurs tests * Add LemmyResult to markdown tests * Add LemmyResult to rate_limiter tests * Add LemmyResult to error tests * Add LemmyResult to api_common utils tests * Add LemmyResult to request tests * Add LemmyResult to claims tests * Propagate registration_applications errors * Remove clippy::unwrap_used from community tests * Add LemmyResult to community_view tests * Add LemmyResult to db_schema post tests * Add LemmyResult to site_aggregates tests * Add LemmyResult to private_message tests * Add LemmyResult to activity tests * Add LemmyResult to federation_allowlist tests * Add LemmyResult to comment_aggregates tests * Add LemmyResult to post_report tests * Add LemmyResult to moderator tests * Add LemmyResult to community_aggregates tests * Add LemmyResult to person_aggregates tests * Add LemmyResult to language tests * Add LemmyResult to post_aggregates tests * Add LemmyResult to db_schema comment tests * Add LemmyResult to actor_language tests * Add LemmyResult to vote_view tests * Add LemmyResult to registration_application_view tests * Add LemmyResult to private_message_view tests * Add LemmyResult to private_message_report_view tests * Add LemmyResult to post_report_view tests * Add LemmyResult to comment_report_view tests * Add LemmyResult to sitemap tests * Replace .expect() with .unwrap() * Format code * Remove clippy::unwrap_used from activity tests * Add diesel result in db_schema tests * Format code * Map to_bytes() error to LemmyErrorType * Remove clippy::unwrap_used from error tests * Removing a few more unwraps, and cleaning up language code. * Replace map_err with unwrap_or_default * Replace ok_or with and_then --------- Co-authored-by: Dessalines <[email protected]> * Adding a get_random_community endpoint. (#5042) * Adding a get_random_community endpoint. - Fixes #4698 * Fixing issue from main. * Adding ListingType to the query. * More concise query filter. * Resolve links to remote posts into local URL (#5057) * move code to new file * rewrite markdown links (fixes #2987) * add missing file * add helper fn * also convert post.url * simplify search.rs * clippy * also rewrite user/community links in markdown * Call from apub handlers, cleanup * no network requests in test * clippy * fix tests * serial * test * no mut * add api test * fix api test * Update Rust crate clap to v4.5.19 (#5080) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * Update Rust crate html2text to v0.12.6 (#5081) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * Update Rust crate rustls to v0.23.14 (#5083) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * Update rust-futures monorepo to v0.3.31 (#5090) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * Update Rust crate actix-web-prom to 0.9.0 (#5091) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * Update Rust crate regex to v1.11.0 (#5093) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * Update Rust crate serde_with to v3.10.0 (#5094) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * Update Rust crate serde_with to v3.11.0 (#5095) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * Update dependency ts-jest to v29.2.5 (#5089) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * Update dependency typescript to v5.6.2 (#5098) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * Update dependency eslint to v9.12.0 (#5097) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * Update docker/dockerfile Docker tag to v1.10 (#5099) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * Update pnpm to v9.12.0 (#5100) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * Update Rust crate reqwest to v0.12.8 (#5082) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * Update dependency @types/node to v22.7.4 (#5096) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * Update tamasfe/taplo Docker tag to v0.9.3 (#5101) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * Update tmknom/prettier Docker tag to v3.2.5 (#5102) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * Add test case for community.hidden in post_view (ref #5074) (#5106) * Resize post thumbnails (fixes #4053) (#5107) * Resize post thumbnails (fixes #4053) * 256px * Update typescript-eslint monorepo to v8.8.1 (#5103) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * Update renovate config (#5109) * Update renovate config * ignroe rust updates, run on first day of month * fix * Add community alphabetic sorting (#5056) * Started * Finished? Need to write tests * Formatting * Formatting * Formatting * Write tests * Formatting * Formatting * Formatting * Unnecessary lifetime * Safety * Unwrap * Formatting * Formatting * Fix local_only test * Formatting * Name consistency * Adding lower to community name sort. --------- Co-authored-by: Dessalines <[email protected]> Co-authored-by: Dessalines <[email protected]> * Support markdown sub/superscript, use external crate for spoilers (#5135) * Use external crate for spoiler tags * Also add other plugins * fix test * Remove comment_like.post_id column which is unnecessary (ref #5122) (#5134) * Allow admins to resolve removed or deleted objects via API (#5061) * Allow admins to resolve removed or deleted objects via API * Removing pointless TestUser. --------- Co-authored-by: Dessalines <[email protected]> * Remove individual user/community inboxes (#5124) * Remove endpoints for individual community/user inboxes fixes #4147 fixes #3928 * Remove shared_inbox_url columns * fmt * Fixing errors. --------- Co-authored-by: Carlos Cabello <[email protected]> Co-authored-by: Nutomic <[email protected]> Co-authored-by: privacyguard <[email protected]> Co-authored-by: privacyguard <[email protected]> Co-authored-by: Freakazoid182 <[email protected]> Co-authored-by: SleeplessOne1917 <[email protected]> Co-authored-by: leoseg <[email protected]> Co-authored-by: Sander Saarend <[email protected]> Co-authored-by: Joseph Silva <[email protected]> Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> Co-authored-by: netbrum <[email protected]> Co-authored-by: Steven Vergenz <[email protected]> Co-authored-by: Richard Schwab <[email protected]> * Search with LemmyErrorType prefix --------- Co-authored-by: Dessalines <[email protected]> Co-authored-by: Carlos Cabello <[email protected]> Co-authored-by: privacyguard <[email protected]> Co-authored-by: privacyguard <[email protected]> Co-authored-by: Freakazoid182 <[email protected]> Co-authored-by: SleeplessOne1917 <[email protected]> Co-authored-by: leoseg <[email protected]> Co-authored-by: Sander Saarend <[email protected]> Co-authored-by: Joseph Silva <[email protected]> Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> Co-authored-by: netbrum <[email protected]> Co-authored-by: Steven Vergenz <[email protected]> Co-authored-by: Richard Schwab <[email protected]> Co-authored-by: Dessalines <[email protected]>

Drop timestamp column from votes (ref #5016)

63a2f7b

Nutomic requested review from dessalines, phiresky, dullbananas and SleeplessOne1917 as code owners October 17, 2024 09:18

fmt

a28d45d

Nutomic changed the title ~~Drop timestamp column from votes (ref #5016)~~ Drop unused vote columns (ref #5016) Oct 17, 2024

Also remove comment_like.post_id

1a0770a

Nutomic mentioned this pull request Oct 17, 2024

Remove redundant field CommentLike.post_id #4749

Closed

restore value of comment_like.post_id

719efb2

set columns null before dropping

019bf67

clean

08055b9

Nutomic marked this pull request as draft October 17, 2024 12:51

add setting store_vote_timestamps

fd8a6b4

Nutomic changed the title ~~Drop unused vote columns (ref #5016)~~ Make vote timestamps optional (ref #5016) Oct 17, 2024

Nutomic marked this pull request as ready for review October 17, 2024 13:57

Nutomic closed this Oct 21, 2024

dessalines reopened this Oct 22, 2024

Nutomic added a commit that referenced this pull request Oct 23, 2024

Remove comment_like.post_id column which is unnecessary (ref #5122)

46b8820

Nutomic closed this Oct 23, 2024

dessalines pushed a commit that referenced this pull request Oct 26, 2024

Remove comment_like.post_id column which is unnecessary (ref #5122) (#…

9258261

…5134)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Make vote timestamps optional (ref #5016) #5122

Make vote timestamps optional (ref #5016) #5122

Nutomic commented Oct 17, 2024 •

edited

Loading

Nothing4You commented Oct 17, 2024

Nutomic commented Oct 17, 2024

Nothing4You commented Oct 17, 2024

Nutomic commented Oct 17, 2024 •

edited

Loading

Pdzly commented Oct 17, 2024

phiresky commented Oct 17, 2024 •

edited

Loading

Nutomic commented Oct 17, 2024 •

edited

Loading

Illecors commented Oct 17, 2024

asimons04 commented Oct 17, 2024 •

edited

Loading

Nutomic commented Oct 17, 2024

dessalines commented Oct 17, 2024

dessalines commented Oct 17, 2024

Nutomic commented Oct 17, 2024

dessalines commented Oct 17, 2024

dullbananas commented Oct 17, 2024

Nutomic commented Oct 18, 2024

Nothing4You commented Oct 19, 2024

Nutomic commented Oct 21, 2024

dessalines commented Oct 22, 2024 •

edited

Loading

Nutomic commented Oct 23, 2024

Make vote timestamps optional (ref #5016) #5122

Make vote timestamps optional (ref #5016) #5122

Conversation

Nutomic commented Oct 17, 2024 • edited Loading

Nothing4You commented Oct 17, 2024

Nutomic commented Oct 17, 2024

Nothing4You commented Oct 17, 2024

Nutomic commented Oct 17, 2024 • edited Loading

Pdzly commented Oct 17, 2024

phiresky commented Oct 17, 2024 • edited Loading

Nutomic commented Oct 17, 2024 • edited Loading

Illecors commented Oct 17, 2024

asimons04 commented Oct 17, 2024 • edited Loading

Nutomic commented Oct 17, 2024

dessalines commented Oct 17, 2024

dessalines commented Oct 17, 2024

Nutomic commented Oct 17, 2024

dessalines commented Oct 17, 2024

dullbananas commented Oct 17, 2024

Nutomic commented Oct 18, 2024

Nothing4You commented Oct 19, 2024

Nutomic commented Oct 21, 2024

dessalines commented Oct 22, 2024 • edited Loading

Nutomic commented Oct 23, 2024

Nutomic commented Oct 17, 2024 •

edited

Loading

Nutomic commented Oct 17, 2024 •

edited

Loading

phiresky commented Oct 17, 2024 •

edited

Loading

Nutomic commented Oct 17, 2024 •

edited

Loading

asimons04 commented Oct 17, 2024 •

edited

Loading

dessalines commented Oct 22, 2024 •

edited

Loading